Text-Guided Graph Neural Networks for Referring 3D Instance Segmentation

This paper addresses a new task called referring 3D instance segmentation, which aims to segment out the target in scene given query sentence. Previous work on understanding has explored visual grounding with natural language guidance, yet emphasis is mostly constrained images and videos. We propose Text-guided Graph Neural Network (TGNN) for segmentation point clouds. Given sentence cloud of scene, our method learns extract per-point features predicts an offset shift each toward its object center. Based offsets, we cluster points produce fused coordinates candidate objects. The resulting clusters are modeled as nodes learn representations that encompass relation structure object. GNN layers leverage object's relations neighbors generate attention heatmap input expression. Finally, used "guide" aggregation information from neighborhood nodes. Our achieves state-of-the-art performance localization ScanRefer, Nr3D, Sr3D benchmarks, respectively.